Challenges of COVID-19 Case Forecasting in the US, 2020-2021 (preprint)

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naive baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making. Author SummaryAs SARS-CoV-2 began to spread throughout the world in early 2020, modelers played a critical role in predicting how the epidemic could take shape. Short-term forecasts of epidemic outcomes (for example, infections, cases, hospitalizations, or deaths) provided useful information to support pandemic planning, resource allocation, and intervention. Yet, infectious disease forecasting is still a nascent science, and the reliability of different types of forecasts is unclear. We retrospectively evaluated COVID-19 case forecasts, which were often unreliable. For example, forecasts did not anticipate the speed of increase in cases in early winter 2020. This analysis provides insights on specific problems that could be addressed in future research to improve forecasts and their use. Identifying the strengths and weaknesses of forecasts is critical to improving forecasting for current and future public health responses.

Resources Required for Implementation of SARS-CoV-2 Screening in Massachusetts K-12 Public Schools in Winter/Spring 2021 (preprint)

Importance CDC guidance emphasizes the importance of in-person education for students in grades kindergarten to 12 (K-12) during the COVID-19 pandemic. CDC encourages weekly SARS-CoV-2 testing of asymptomatic, unvaccinated students and staff ("screening") to reduce infection risk and provide data about in-school SARS-CoV-2 prevalence where community incidence is high. The financial costs of screening assays have been described, but the human resource requirements at the school and district level to implement a SARS-CoV-2 screening program are not well known. Objective To quantify the resources required to implement a screening program in K-12 schools. Design, Setting, and Participants A consortium of Massachusetts public K-12 schools was formed to implement and evaluate a range of SARS-CoV-2 screening approaches. Participating districts were surveyed weekly about their programs, including: type of assay used, individual vs. pooled screening, approaches to return of results and deconvolution (identification of positive individual specimens) of positive pools, number and type of personnel implementing the screening program, and hours spent on program implementation. Main Outcomes and Measures Costs, resource utilization Results In 21 participating districts, over 21 weeks from January to June 2021, the positivity rate was 0.0%-0.21% among students and 0.0%-0.13% among educators/staff, and 4 out of 21 (19%) districts had at least one classroom transition to remote learning at any point due to a positive case. The average weekly cost to implement a screening program, including assay and personnel costs, was $17.00 per person tested; this was $46.68 for individual screenings and $15.61 for pooled screenings. The total weekly costs by district ranged from $1,644-$93,486, and districts screened between 58 and 3,675 people per week. The reported number of personnel working per week ranged from 1-5 to >50, and the total number of hours worked by all personnel ranged from 5-10 to >50. Conclusion and Relevance The human resources required to implement SARS-CoV-2 screening in Massachusetts public K-12 schools were substantial. Where screening is recommended for the 2021-22 school year due to high COVID-19 incidence (e.g., where vaccination uptake is low and/or more infectious variants predominate), understanding the human resources required to implement screening will assist districts policymakers in planning.

When do elementary students need masks in school? Model-estimated risk of in-school SARS-CoV-2 transmission and related infections among household members before and after student vaccination (preprint)

BackgroundWhile CDC guidance for K-12 schools recommends indoor masking regardless of vaccination status, final decisions about masking in schools will be made at the local and state level. The impact of the removal of mask restrictions, however, on COVID-19 outcomes for elementary students, educators/staff, and their households is not well known. MethodsWe used a previously published agent-based dynamic transmission model of SARS-CoV-2 in K-12 schools to simulate an elementary school with 638 students across 6 scenarios: combinations of three viral infectiousness levels (reflecting wild-type virus, alpha variant, and delta variant) and two student vaccination levels (0% and 50% coverage to reflect potential authorization in this age group). For each scenario, we varied observed community COVID-19 incidence (0 to 50 cases/100,000 people/day) and mitigation effectiveness (0-100% reduction to in-school secondary attack rate), and evaluated two outcomes over a 30 day period: (1) the probability of at least one in-school transmission, and (2) the increase in total cases among students, educators/staff, and their household members between in-person and remote instruction. ResultsOver 30 days in the simulated elementary school, the probability of at least one in-school SARS-CoV-2 transmission and the number of projected infections in the immediate school community varied widely. In one scenario with the delta variant and no student vaccination, assuming that baseline mitigation measures of simple ventilation and handwashing reduce the secondary attack rate by 40%, if decision-makers seek to keep the monthly probability of an in-school transmission below 50%, additional mitigation (e.g., masking) would need to be added at a community incidence of approximately 4/100,000/day. Once students are vaccinated, thresholds shift substantially higher. LimitationsThe interpretation of model results should be limited by the uncertainty in many of the parameters, including the effectiveness of individual mitigation interventions and vaccine efficacy against the delta variant, and the limited scope of the model beyond the school community. Additionally, the assumed case detection rate (33% of cases detected) may be too high in areas with decreased testing capacity. ConclusionDespite the assumption of high adult vaccination, the risks of both in-school SARS-CoV-2 transmission and resulting infections among students, educators/staff, and their household members remain high when the delta variant predominates and students are unvaccinated. Mitigation measures or vaccinations for students when available can substantially reduce these risks. These findings underscore the potential role for responsive plans, where mitigation is deployed based on local COVID-19 incidence and vaccine uptake.

The Effect of NFL and NCAA Football Games on the Spread of COVID-19 in the United States: An Empirical Analysis (preprint)

ImportanceIn 2020 and early 2021, the National Football League (NFL) and National Collegiate Athletic Association (NCAA) had opted to host games in stadiums across the country. The in-person attendance of games has varied with time and from county to county. There is currently no evidence on whether limited in-person attendance of games has caused a substantial increase in coronavirus disease 2019 (COVID-19) cases. ObjectiveTo assess whether NFL and NCAA football games with limited in-person attendance have contributed to a substantial increase in COVID-19 cases in the counties they were held. DesignIn this time-series cross-sectional study, we matched every county hosting game(s) with in-person attendance (treated) in 2020 and 2021 with a county that has an identical game history for up to 14 days (control). We employed a standard matching method to further refine this matched set so that the treated and matched control counties have similar population size, non-pharmaceutical intervention(s) in place, and COVID-19 trends. We assessed the effect of hosting games with in-person attendance using a difference-in-difference estimator. SettingU.S. counties. ParticipantsU.S. counties hosting NFL and/or NCAA games. ExposureHosting NFL and/or NCAA games. Main outcomes and measuresEstimating the impact of NFL and NCAA games with in-person attendance on new, reported COVID-19 cases per 100,000 residents at the county-level up to 14 days post-game. ResultsThe matching algorithm returned 361 matching sets of counties. The effect of in-person attendance at NFL and NCAA games on community COVID-19 spread is not significant as it did not surpass 5 new daily cases of COVID-19 per 100,000 residents on average. Conclusions and relevanceThis time-series, cross-sectional matching study with a difference-in-differences design did not find an increase in COVID-19 cases per 100,000 residents in the counties where NFL and NCAA games were held with in-person attendance. Our study suggests that NFL and NCAA football games hosted with limited in-person attendance do not cause a significant increase in local COVID-19 cases. Key pointsO_ST_ABSQuestionC_ST_ABSDid NFL and NCAA football games with limited in-person attendance cause a substantia increase in coronavirus disease 2019 (COVID-19) cases in the U.S. counties where the games were held? FindingsThis time-series, cross-sectional study of U.S. counties with NFL and NCAA football games used matching and difference-in-differences design to estimate the effect of games with limited in-person attendance on county-level COVID-19 spread. Our study does not find an increase in county-level COVID-19 cases per 100,000 residents due to NFL and NCAA football games held with limited in-person attendance. MeaningThis study suggests that NFL and NCAA games held with limited in-person attendance do not cause an increase in COVID-19 cases in the counties they are held.

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US (preprint)

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. In 2020, the COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized hundreds of thousands of specific predictions from more than 50 different academic, industry, and independent research groups. This manuscript systematically evaluates 23 models that regularly submitted forecasts of reported weekly incident COVID-19 mortality counts in the US at the state and national level. One of these models was a multi-model ensemble that combined all available forecasts each week. The performance of individual models showed high variability across time, geospatial units, and forecast horizons. Half of the models evaluated showed better accuracy than a naive baseline model. In combining the forecasts from all teams, the ensemble showed the best overall probabilistic accuracy of any model. Forecast accuracy degraded as models made predictions farther into the future, with probabilistic accuracy at a 20-week horizon more than 5 times worse than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. f